<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                 "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <meta http-equiv="Content-type" content="text/html;charset=UTF-8">
  <title>The deal.II Testsuite</title>
  <link href="../screen.css" rel="StyleSheet">
  <meta name="copyright" content="Copyright (C) 1998 - 2020 by the deal.II Authors">
  <meta name="keywords" content="deal dealii finite elements fem triangulation">
  <meta http-equiv="content-language" content="en">
</head>
<body>


    <h1>The deal.II Testsuite</h1>

    <p>
      deal.II has a testsuite with thousands of small programs that we run every time
      we make a change to make sure
      that no existing functionality is broken. It is located in the <code>tests/</code>
      directory of the development repository. The expected output for
      every test is stored in an <code>*.output</code> next to the <code>.cc</code> file,
      and when running a test, you are notified if the output differs from the expected
      output.
    </p>

    <p>
      These days, every time we add a significant piece of functionality,
      we add at least one new test to the testsuite, and we also do so if
      we fix a bug, in both cases to make sure that future changes do not
      break this functionality (again). Machines running the testsuite
      submit the results to our cdash
      <a href="https://cdash.43-1.org/index.php?project=deal.II">CDash instance</a>,
      a webpage showing the status of our regression tests.
    </p>

    <div class="toc">
      <ol>
        <li><a href="#setup">Setting up the testsuite</a></li>
        <ol>
          <li><a href="#setupbuild">For a build directory</a></li>
          <li><a href="#setupinstalled">For an already installed library</a></li>
        </ol>
        <li><a href="#run">Running the testsuite</a></li>
        <ol>
          <li><a href="#runoutput">How to interpret the output</a></li>
          <li><a href="#coverage">Generating coverage information</a></li>
        </ol>
        <li><a href="#layout">Testsuite development</a></li>
        <ol>
          <li><a href="#layoutgeneral">General layout</a></li>
          <li><a href="#restrictbuild">Restricting tests to build configurations</a></li>
          <li><a href="#restrictfeature">Restricting tests to feature configurations</a></li>
          <li><a href="#mpi">Running tests with MPI</a></li>
          <li><a href="#binary">Tests with binary output</a></li>
          <li><a href="#variants">Tests with multiple comparison files</a></li>
          <li><a href="#expect">Changing condition for success</a></li>
          <li><a href="#layoutaddtests">Adding new tests</a></li>
          <li><a href="#layoutaddcategory">Adding new categories</a></li>
        </ol>
        <li><a href="#submit">Submitting test results</a></li>
        <li><a href="#build_tests">Build tests</a></li>
          <ol>
            <li><a href="#dedicatedbuilds">Dedicated build tests</a></li>
          </ol>
      </ol>
    </div>

    <a name="setup"></a>
    <h2>Setting up the testsuite</h2>

    <p>
      The testsuite is part of the development sources of deal.II and located
      under the <code>tests</code> subdirectory. The easiest way to obtain
      both of them is to check out the current development sources via git:
<pre>
$ git clone https://github.com/dealii/dealii
</pre>
    </p>

    <a name="setupbuild"></a>
    <h3>For a build directory</h3>

    <p>
      To enable the testsuite for a given build directory, ensure that
      deal.II is successfully configured and built (installation is not
      necessary). After that you can set up the testsuite via the
      "setup_tests" target:
<pre>
$ make setup_tests
</pre>
      This will set up all tests supported by the current configuration.
      The testsuite can now be run in the current <i>build directory</i> as
      described below.
    </p>

    <p>
      The setup can be fine-tuned using the following commands:
<pre>

$ make prune_tests - removes all testsuite subprojects
</pre>

    <p>
      In addition, when setting up the testsuite, the following environment
      variables can be used to override default behavior when
      calling <code>make setup_tests</code>:
<pre>
TEST_TIME_LIMIT
  - The time limit (in seconds) a single test is allowed to take. Defaults
    to 180 seconds

TEST_PICKUP_REGEX
  - A regular expression to select only a subset of tests during setup.
    An empty string is interpreted as a catchall (this is the default).
</pre>
For example,
<pre>
TEST_PICKUP_REGEX="umfpack" make setup_tests
</pre>
will only enable tests which match the string "umfpack" in category or
name.
    </p>

    <a name="setupinstalled"></a>
    <h3>For an already installed library</h3>

    <p>
      The testsuite can also be set up for an already installed library
      (starting with version 8.3). For this, create a build directory for
      the testsuite and run cmake pointing to the <code>tests</code>
      subdirectory, e.g.,
<pre>
$ mkdir tests_for_installed_dealii
$ cd tests_for_installed_dealii
$ cmake -DDEAL_II_DIR=/path/to/installed/dealii /path/to/dealii_source/tests
</pre>
      After that the same configuration targets as described <a
      href="setupbuild">above</a> are available.
    </p>

    <a name="run"></a>
    <h2>Running the testsuite</h2>

    <p>
      The testsuite can now be run in the <i>build directory</i> via
<pre>
$ ctest [-j N]
</pre>
      Here, <code>N</code> is the number of concurrent tests that should be
      run, in the same way as you can say <code>make -jN</code>. The testsuite
      is huge and will need around 12h on current computers
      running single threaded.
    </p>

    <p>
      If you only want to run a subset of tests
      matching a regular expression, or if you want to exclude tests matching
      a regular expression, you can use
<pre>
$ ctest [-j N] -R '&lt;positive regular expression&gt;'
$ ctest [-j N] -E '&lt;negative regular expression&gt;'
</pre>
    </p>

    <p>
      <b>Note:</b>
      Not all tests succeed on every machine even if all computations are
      correct, because your machine generates slightly different floating
      point outputs. To increase the number of tests that work correctly,
      install the
      <a href="http://www.nongnu.org/numdiff/">numdiff</a> tool that compares
      stored and newly created output files based on floating point
      tolerances. To use it, simply export where the <code>numdiff</code>
      executable can be found via the <code>PATH</code>
      environment variable so that it can be found during
      <code>make setup_tests</code>.
    </p>

    <a name="runoutput"></a>
    <h3>How to interpret the output</h3>

    <p>
      A typical output of a <code>ctest</code> invocation looks like:
<pre>
$ ctest -j4 -R "base/thread_validity"
Test project /tmp/trunk/build
      Start 747: base/thread_validity_01.debug
      Start 748: base/thread_validity_01.release
      Start 775: base/thread_validity_05.debug
      Start 776: base/thread_validity_05.release
 1/24 Test #776: base/thread_validity_05.release ...   Passed    1.89 sec
 2/24 Test #748: base/thread_validity_01.release ...   Passed    1.89 sec
      Start 839: base/thread_validity_03.debug
      Start 840: base/thread_validity_03.release
 3/24 Test #747: base/thread_validity_01.debug .....   Passed    2.68 sec
[...]
      Start 1077: base/thread_validity_08.debug
      Start 1078: base/thread_validity_08.release
16/24 Test #1078: base/thread_validity_08.release ...***Failed    2.86 sec
18/24 Test #1077: base/thread_validity_08.debug .....***Failed    3.97 sec
[...]

92% tests passed, 2 tests failed out of 24

Total Test time (real) =  20.43 sec

The following tests FAILED:
        1077 - base/thread_validity_08.debug (Failed)
        1078 - base/thread_validity_08.release (Failed)
Errors while running CTest
</pre>
      If a test failed (like <code>base/thread_validity_08.debug</code> in above
      example output), you might want to find out what exactly went wrong. To
      this end, you can search
      through <code>Testing/Temporary/LastTest.log</code> for the exact output
      of the test, or you can rerun this one test, specifying <code>-V</code>
      to select verbose output of tests:
<pre>
$ ctest -V -R "base/thread_validity_08.debug"
[...]
test 1077
    Start 1077: base/thread_validity_08.debug

1077: Test command: [...]
1077: Test timeout computed to be: 600
1077: Test base/thread_validity_08.debug: RUN
1077: ===============================   OUTPUT BEGIN  ===============================
1077: Built target thread_validity_08.debug
1077: Generating thread_validity_08.debug/output
1077: terminate called without an active exception
1077: /bin/sh: line 1: 18030 Aborted [...]/thread_validity_08.debug
1077: base/thread_validity_08.debug: BUILD successful.
1077: base/thread_validity_08.debug: RUN failed. Output:
1077: DEAL::OK.
1077: gmake[3]: *** [thread_validity_08.debug/output] Error 1
1077: gmake[2]: *** [CMakeFiles/thread_validity_08.debug.diff.dir/all] Error 2
1077: gmake[1]: *** [CMakeFiles/thread_validity_08.debug.diff.dir/rule] Error 2
1077: gmake: *** [thread_validity_08.debug.diff] Error 2
1077:
1077:
1077: base/thread_validity_08.debug: ******    RUN failed    *******
1077:
1077: ===============================    OUTPUT END   ===============================
</pre>
      So this specific test aborted in the <code>RUN</code> stage.
    </p>

    <p>
      The general output for a successful test <code>&lt;test&gt;</code> in
      category <code>&lt;category&gt;</code> for build type
      <code>&lt;build&gt;</code> is
<pre>
xx: Test &lt;category&gt;/&lt;test&gt;.&lt;build&gt;: PASSED
xx: ===============================   OUTPUT BEGIN  ===============================
xx: [...]
xx: &lt;category&gt;/&lt;test&gt;.&lt;build&gt;: PASSED.
xx: ===============================    OUTPUT END   ===============================
</pre>
      And for a test that fails in stage <code>&lt;stage&gt;</code>:
<pre>
xx: Test &lt;category&gt;/&lt;test&gt;.&lt;build&gt;: &lt;stage&gt;
xx: ===============================   OUTPUT BEGIN  ===============================
xx: [...]
xx: &lt;category&gt;/&lt;test&gt;.&lt;build&gt;: &lt;stage&gt; failed. [...]
xx:
xx: &lt;category&gt;/&lt;test&gt;.&lt;build&gt;: ******    &lt;stage&gt; failed    *******
xx: ===============================    OUTPUT END   ===============================
</pre>
      Hereby, <code>&lt;stage&gt;</code> indicates the stage in which the
      test failed:
      <ul>
        <li>
          <code>CONFIGURE</code>: only for test in the "build_tests"
          category: The test project failed in the configuration stage
        </li>
        <li>
          <code>BUILD</code>: compilation error occurred
        </li>
        <li>
          <code>RUN</code>: the test executable could not be run / aborted
        </li>
        <li>
          <code>DIFF</code>: the test output differs from the reference output
        </li>
        <li>
          <code>PASSED</code>: the test run successful
        </li>
      </ul>
      Typically, tests fail because the output has changed, and you will see
      this in the DIFF phase of the test.
    </p>


    <a name="memcheck"></a>
     <h3>Running with valgrind</h3>

     <p>
       The testsuite can also be run using <a href="http://valgrind.org/">valgrind</a>
       to check for <i>memory corruption</i> in the library.
     </p>

     <p>
       You can do so by invoking
 <pre>
 ctest -DMEMORYCHECK=ON <...> -S ../tests/run_testsuite.cmake
 </pre>
       when running the testsuite, or directly by
 <pre>
 ctest <...> -S ../tests/run_memorycheck.cmake
 </pre>
     </p>

     <p>
       At the end of all of this, results will be shown in a separate section
       "Dynamic Analysis" at the
       <a href="https://cdash.43-1.org/index.php?project=deal.II&display=project"
       target="_top">deal.II cdash site</a>.
     </p>


    <a name="coverage"></a>
    <h3>Generating coverage information</h3>

    <p>
      The testsuite can also be used to provide <i>coverage</i> information,
      i.e., data that shows which lines of the library are executed how many
      times by running through all of the tests in the testsuite. This is of
      interest in finding places in the library that are not covered by
      the testsuite and, consequently, are prone to the inadvertent
      introduction of bugs since existing functionality is not subject to
      existing tests.
    </p>

    <p>
      To run the testsuite in this mode, essentially, you have to do three
      things:
      <ol>
	<li>
	  build the library with appropriate profiling flags
	</li>
	<li>
	  run all or some tests (built with the same profiling flags)
	</li>
	<li>
	  gather all information and convert them to a viewable format.
	</li>
      </ol>
      In order to achieve the first two, configure the library with
<pre>
cmake -DCMAKE_BUILD_TYPE=Debug -DDEAL_II_SETUP_COVERAGE=ON <...>
</pre>
      You can then build the library and run the tests as usual.
    </p>

    <p>
      For the last point, one can in principal use whatever tool one
      wants. That said, the deal.II ctest driver already has builtin
      functionality to gather all profiling files and submit them to cdash
      where we already gather testsuite results
      (see <a href="#submit">below</a>). You can do so by invoking
<pre>
ctest -DCOVERAGE=ON <...> -S ../tests/run_testsuite.cmake
</pre>
      when running the testsuite, or directly by
<pre>
ctest <...> -S ../tests/run_coverage.cmake
</pre>
    </p>

    <p>
      At the end of all of this, results will be shown in a separate section
      "Coverage" at the
      <a href="https://cdash.43-1.org/index.php?project=deal.II&display=project"
      target="_top">deal.II cdash site</a>.
      In case you download the coverage report uploader for Codecov via
      <code>contrib/utilities/download_codecov</code>, the coverage
      report will also be uploaded to the
      <a href="https://codecov.io/gh/dealii/dealii"
      target="_top">Codecov dashboard</a>.
    </p>


    <a name="layout"></a>
    <h2>Testsuite development</h2>

    <p>
      The following outlines what you need to know if you want to understand
      how the testsuite actually works, for example because you may want to
      add tests along with the functionality you are currently developing.
    </p>



    <a name="layoutgeneral"></a>
    <h3>General layout</h3>

    <p>
      A test usually consists of a source file and an output file for
      comparison (under the testsuite directory <code>tests</code>):
<pre>
category/test.cc
category/test.output
</pre>
      <code>category</code> will be one of the existing subdirectory
      under <code>tests/</code>, e.g., <code>lac/</code>, <code>base/</code>,
      or <code>mpi/</code>. Historically, we have grouped tests into the
      directories <code>base/, lac/, deal.II/</code> depending on their
      functionality, and <code>bits/</code> if they were small unit tests, but
      in practice we have not always followed this rigidly. There are also
      more specialized directories <code>trilinos/, petsc/,
      serialization/, mpi/</code> etc, whose meaning is more obvious.
      <code>test.cc</code> must be a regular executable (i.e. having an
      <code>int main()</code> routine). It will be compiled, linked and
      run. The executable should not output anything to <code>cout</code>
      (at least under normal circumstances, i.e. no error condition),
      instead the executable should output to a file <code>output</code>
      in the current working directory. In practice, we rarely write the
      source files completely from scratch, but we find an existing test that
      already does something similar and copy/modify it to fit our needs.
    </p>
    <p>
      For a normal test, <code>ctest</code> will typically run the following 3
      stages:
      <ul>
        <li>
          <code>BUILD</code>: The build stage generates an executable in
          <code>BUILD_DIR/tests/&lt;category&gt;/&lt;test&gt;</code>.
        </li>
        <li>
          <code>RUN</code>: The run stage then invokes the executable in
          the directory where it is located. By convention, each test
          either prints its test results directly to <code>stdout</code>,
          or directly into a file called <code>output</code> (in the
          current working directory). The latter takes precedence.
          The output (via stdout or file) will then be located in
          <code>BUILD_DIR/tests/&lt;category&gt;/&lt;test&gt;/output</code>.
          If the run fails (e.g. because the program aborts with an error
          code) the file <code>output</code> is renamed to
          <code>failing_output</code>.
        </li>
        <li>
          <code>DIFF</code>: As a last stage the generated output file will
          be compared to
          <code>SOURCE_DIR/tests/&lt;category&gt;/&lt;test&gt;[...].output</code>.
          and stored in
          <code>BUILD_DIR/tests/&lt;category&gt;/&lt;test&gt;/diff</code>.
          If the diff fails  the file <code>diff</code> is renamed to
          <code>failing_diff</code>.
        </li>
      </ul>
    </p>


    <a name="restrictbuild"></a>
    <h3>Restricting tests to build configurations</h3>

    <p>
      Comparison file can actually be named in a more complex way than
      just <code>category/test.output</code>. In pseudo code:
<pre>
category/test.[with_&lt;string&gt;(&lt;=|&gt;=|=|&lt;|&gt;)&lt;on|off|version&gt;.]*
              [mpirun=&lt;x&gt;.][expect=&lt;y&gt;.][binary.][&lt;debug|release&gt;.]output
</pre>
      Normally, a test will be set up so that it runs twice, once in debug and
      once in release configuration.
      If a specific test can only be run in debug or release configurations but
      not in both it is possible to restrict the setup by prepending
      <code>.debug</code> or <code>.release</code> directly before
      <code>.output</code>, e.g.:
<pre>
category/test.debug.output
</pre>
      This way, the test will only be set up to build and run against the debug
      library. If a test should run in both configurations but, for some
      reason, produces different output (e.g., because it triggers an
      assertion in debug mode), then you can just provide two different output
      files:
<pre>
category/test.debug.output
category/test.release.output
</pre>
    </p>

    <a name="restrictfeature"></a>
    <h3>Restricting tests to feature configurations</h3>
    <p>
      In a similar vain as for build configurations, it is possible to restrict
      tests to specific feature configurations, e.g.,
<pre>
category/test.with_umfpack=on.output, or
category/test.with_zlib=off.output
</pre>
      These tests will only be set up if the specified feature was configured.
      It is possible to provide different output files for disabled/enabled
      features, e.g.,
<pre>
category/test.with_64bit_indices=on.output
category/test.with_64bit_indices=off.output
</pre>
      Furthermore, a test can be restricted to be run only if specific
      versions of a feature are available. For example
<pre>
category/test.with_trilinos.geq.11.14.1.output
</pre>
      will only be run if (a) trilinos is available, i.e.,
      <code>DEAL_II_WITH_TRILINOS=TRUE</code> and (b) if trilinos is at least
      of version 11.14.1. Supported operators are
      <code>=</code>, <code>.le.;</code>, <code>.ge.</code>, <code>.leq.</code>,
      <code>.geq.</code>.
    </p>
    <p>
      It is also possible to declare multiple constraints subsequently, e.g.
<pre>
category/test.with_umfpack=on.with_zlib=on.output
</pre>
    </p>
    <p>
      <b>Note:</b> The tests in some subdirectories of <code>tests/</code> are
      automatically run only if some feature is enabled. In this case a
      feature constraint encoded in the output file name is redundant and
      should be avoided. In particular, this holds for subdirectories
      <code>distributed_grids</code>, <code>lapack</code>,
      <code>metis</code>, <code>petsc</code>, <code>slepc</code>,
      <code>trilinos</code>, <code>umfpack</code>, <code>gla</code>, and
      <code>mpi</code>
    </p>


    <a name="mpi"></a>
    <h3>Running tests with MPI</h3>
    <p>
      If a test should be run with MPI in parallel, the number of MPI
      processes <code>N</code> with which a program needs to be run for
      comparison with a given output file is specified as follows:
<pre>
category/test.mpirun=N.output
</pre>
      It is quite typical for an MPI-enabled test to have multiple output
      files for different numbers of MPI processes.
    </p>

    <a name="variants"></a>
    <h3>Tests with multiple comparison files</h3>
    <p>
      Sometimes it is necessary to provide multiple comparison files for a
      single test, for example because you want to test code on multiple
      platforms that produce different output files that, nonetheless, all
      should be considered correct. An example would be tests that use the
      <code>rand()</code> function that is implemented differently on
      different platforms. Additional comparison files have the same path
      as the main comparison file (in this case <code>test.output</code>)
      followed by a dot and a variant description:
<pre>
category/test.output
category/test.output.2
category/test.output.3
category/test.output.4
</pre>
      The testsuite will try to match the output against all variants in
      alphabetical order starting with the main output file.
    </p>

    <p>
      <b>Warning:</b> This mechanism is only meant as a last resort for
      tests where <i>no</i> alternative approach is viable. Especially,
      consider first to
      <ol>
        <li> make the test more robust such that differences can be
          expressed in round-off errors detectable by numdiff.
        <li> restrict comparison files to specific
          <a href="#restrictfeature">versions of an external feature</a>.
      </ol>
    </p>

    <p>
      <b>Note:</b> The main comparison file (i.e., the one ending in
      <code>output</code> is mandatory. Otherwise, no test will be
      configured.
    </p>


    <a name="expect"></a>
    <h3>Changing condition for success</h3>
    <p>
      Normally a test is considered to be successful if all test stages
      could be run and the test reached the <code>PASSED</code> stage (see
      the <a href="#runoutput">output description</a> section for details).
      If (for some reason) the test should succeed ending at a specific
      test stage different than <code>PASSED</code> you can specify it via
      <code>expect=&lt;stage&gt;</code>, e.g.:
<pre>
category/test.expect=run.output
</pre>
    </p>


    <a name="layoutaddtests"></a>
    <h3>Adding new tests</h3>

    <p>
      We typically add one or more new tests every
      time we add new functionality to the library or fix a bug. If you
      want to contribute code to the library, you should do this
      as well. Here's how: you need a testcase and a file with the
      expected output.
    </p>

    <h4>The testcase</h4>
    <p>
      For the testcase, we usually start from one of the existing tests, copy
      and modify it to where it does what we'd like to test. Alternatively,
      you can also start from a template like this:
<pre>
// ---------------------------------------------------------------------
//
// Copyright (C) 2015 - 2020 by the deal.II Authors
//
// This file is part of the deal.II library.
//
// The deal.II library is free software; you can use it, redistribute
// it, and/or modify it under the terms of the GNU Lesser General
// Public License as published by the Free Software Foundation; either
// version 2.1 of the License, or (at your option) any later version.
// The full text of the license can be found in the file LICENSE.md at
// the top level directory of deal.II.
//
// ---------------------------------------------------------------------

// a short (a few lines) description of what the program does

#include "../tests.h"

// all include files you need here

int main ()
{
  // Initialize deallog for test output.
  // This also reroutes deallog output to a file "output".
  initlog();

  // your testcode here:
  int i = 0;
  deallog << i << std::endl;

  return 0;
}
</pre>

    <p>This code opens an output file <code>output</code> in the current working
    directory and then writes all output you generate to it, through the
    <code>deallog</code> stream. The <code>deallog</code> stream works like
    any other <code>std::ostream</code> except that it does a few more
    things behind the scenes that are helpful in this context. In above
    case, we only write a zero to the output file. Most tests of course
    write computed data to the output file to make sure that whatever we
    compute is what we got when the test was first written.
    </p>

    <p>
    There are a number of directories where you can put a new test.
    Extensive tests of individual classes or groups of classes
    have traditionally been into the <code>base/</code>,
    <code>lac/</code>, <code>deal.II/</code>, <code>fe/</code>,
    <code>hp/</code>, or <code>multigrid/</code> directories, depending on
    where the classes that are tested are located. More atomic tests often go
    into <code>bits/</code>. There are also
    directories for PETSc and Trilinos wrapper functionality.
    </p>

    <h4>An expected output</h4>

    <p>
      In order to run your new test, copy it to an appropriate category and
      create an empty comparison file for it:
<pre>
category/my_new_test.cc
category/my_new_test.output
</pre>
      Now, rerun
<pre>
$ make setup_tests
</pre>
      so that your new test is picked up. After that it is possible to
      invoke it with
<pre>
$ ctest -V -R "category/my_new_test"
</pre>
    </p>

    <p>
      If you run your new test executable this way, the test should compile
      and run successfully but fail in the diff stage (because of the empty
      comparison file). You will get an output file
      <code>BUILD_DIR/category/my_new_test/output</code>. Take a look at it to
      make sure that the output is what you had expected. (For complex tests,
      it may sometimes be impossible to say whether the output is correct, and
      in this case we sometimes just take it to make
      sure that future invocations of the test yield the same results.)
    </p>

    <p>
      The next step is to copy and rename this output file to the source
      directory and replace the original comparison file with it:
<pre>
category/my_new_test.output
</pre>
      At this point running the test again should be successful:
<pre>
$ ctest -V -R "category/my_new_test"
</pre>
    </p>

    <a name="layoutaddcategory"></a>
    <h2>Adding new categories</h2>

    <p>
      If you want to create a new category in the testsuite, create an new
      folder under <cmake>./tests</cmake> that is named accordingly and put
      a <code>CMakeLists.txt</code> file into it containing
    </p>
<pre>
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.12)
INCLUDE(../setup_testsubproject.cmake)
PROJECT(testsuite CXX)
INCLUDE(${DEAL_II_TARGET_CONFIG})
DEAL_II_PICKUP_TESTS()
</pre>

    <a name="submit"></a>
    <h2>Submitting test results</h2>

    <p>
      To submit test results to our <a
        href="https://cdash.43-1.org/index.php?project=deal.II">CDash</a>
      instance just invoke ctest within a build directory (or designated
      build directory) with the <code>-S</code> option pointing to the
      <code>run_testsuite.cmake</code> script (assuming here for the
      moment that your build directory is a sub-directory of the
      source directory):
<pre>
$ ctest [...] -V -S ../tests/run_testsuite.cmake
</pre>
      The script will run configure, build and ctest and submit the results
      to the CDash server. It does not matter whether the configure, build
      or ctest stages were run before that. Also in script mode, you can
      specify the same options for <code>ctest</code> as explained above.
    </p>

    <p>
      <b>Note:</b>
      For also building the tests in parallel you have to provide suitable flags
      using MAKEOPTS as well, i.e. you would typically use
<pre>
$ ctest -DMAKEOPTS="-j N" -j N [...] -V -S ../tests/run_testsuite.cmake
</pre>
      for compiling the library and running the tests in parallel,
      where <code>N</code> is the number of jobs you want to
      simultaneously execute on your machine; you would typically
      choose <code>N</code> equal to the number of processor cores you
      have in your machine, or maybe somewhere around one half to
      three quarters if you don't want to overload it or are short on
      memory.
    </p>

    <p>
      It is possible to run tests and submit results for an already installed library by
<pre>
mkdir build && cd build
cp $DEAL_II_SOURCE_DIR/CTestConfig.cmake .
ctest \
  -DCTEST_SOURCE_DIRECTORY=$DEAL_II_SOURCE_DIR/tests \
  -DDEAL_II_DIR=$DEAL_II_DIR \
  [...] -S $DEAL_II_SOURCE_DIR/tests/run_testsuite.cmake -V
</pre>
    </p>

    <p>
      <b>Note:</b> The default output in script mode is very minimal.
      Therefore, it is recommended to specify <code>-V</code> which will
      give the same level of verbosity as the non-script mode.
    </p>

    <p>
      <b>Note:</b> The following variables can be set to via
<pre>
ctest -D&lt;variable&gt;=&lt;value&gt; [...]
</pre>
      to control the behaviour of the <code>run_testsuite.cmake</code>
      script:
<pre>
CTEST_SOURCE_DIRECTORY
  - The source directory of deal.II
  - If unspecified, "../" relative to the location of this script is
    used. If this is not a source directory, an error is thrown.

CTEST_BINARY_DIRECTORY
  - The designated build directory (already configured, empty, or non
    existent - see the information about TRACKs what will happen)
  - If unspecified the current directory is used. If the current
    directory is equal to CTEST_SOURCE_DIRECTORY or the "tests"
    directory, an error is thrown.

CTEST_CMAKE_GENERATOR
  - The CMake Generator to use (e.g. "Unix Makefiles", or "Ninja", see
    $ man cmake)
  - If unspecified the generator of a configured build directory will
    be used, otherwise "Unix Makefiles".

TRACK
  - The track the test should be submitted to. Defaults to
    "Experimental". Possible values are:

    "Experimental"     - all tests that are not specifically "build" or
                         "regression" tests should go into this track

    "Build Tests"      - Build tests that configure and build in a clean
                         directory (without actually running the
                         testsuite)

    "Regression Tests" - Reserved for the "official" regression tester

    "Continuous"       - Reserved for the "official" regression tester

CONFIG_FILE
  - A configuration file (see doc/users/config.sample)
    that will be used during the configuration stage (invokes
    $ cmake -C ${CONFIG_FILE}). This only has an effect if
    CTEST_BINARY_DIRECTORY is empty.

DESCRIPTION
  - A string that is appended to CTEST_BUILD_NAME

COVERAGE
  - If set to ON deal.II will be configured with
  DEAL_II_SETUP_COVERAGE=ON, CMAKE_BUILD_TYPE=Debug and the
  CTEST_COVERAGE() stage will be run. Test results must go into the
  "Experimental" section.

MAKEOPTS
  - Additional options that will be passed directly to make (or ninja).
</pre>

      Furthermore, the variables TEST_TIME_LIMIT and TEST_PICKUP_REGEX (as
      described <a href="#setupbuild">above</a>), DIFF_DIR, and NUMDIFF_DIR
      can also be set and will be handed automatically down
      to <code>cmake</code>. For more details on the different tracks, see
      the <a href="https://github.com/dealii/dealii/wiki/Testing-Infrastructure">
      Testing Infrastructure Wiki page</a>.
    </p>



    <a name="build_tests"></a>
    <h2>Build tests</h2>

    <p>
      Build tests are used to check that deal.II can be compiled on
      different systems and with different compilers as well as
      different configuration options. Results are collected in the "Build
      Tests" track in <a
      href="https://cdash.43-1.org/index.php?project=deal.II">CDash</a>.<p>

      <p>Running the build test suite is simple and we encourage deal.II
      users with configurations not found on the <a
      href="https://cdash.43-1.org/index.php?project=deal.II">CDash page</a> to
      participate. Assuming you checked out deal.II into the directory
      <code>dealii</code>, running it is as simple as:
<pre>
mkdir dealii/build
cd dealii/build
ctest -j4 -S ../tests/run_buildtest.cmake
</pre>
    </p>

    <p>
      What this does is to compile and build deal.II in the directory
      <code>build</code> (which includes building all configurable tutorial
      programs as well) but does not run the full testsuite. The results
      are sent to the CDash instance.
    </p>

    <p>
      <b>Note:</b> Build tests require the designated build directory to be
      completely empty. If you want to specify a build configuration for
      cmake use a <a href="../users/config.sample">configuration file</a>
      to preseed the cache as explained <a href="#submit">above</a>:
<pre>
$ ctest -DCONFIG_FILE="[...]/config.sample" [...]
</pre>
    </p>



    <a name="dedicatedbuilds"></a>
    <h3>Dedicated build tests</h3>

    <p>
      Build tests work best if they run automatically and periodically.
      There is a detailed example for such dedicated build tests on the <a
        href="https://github.com/dealii/dealii/wiki/Build-Tests">wiki</a>.
    </p>


    <hr />
    <div class="right">
      <a href="http://validator.w3.org/check?uri=referer" target="_top">
        <img style="border:0" src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!"></a>
      <a href="http://jigsaw.w3.org/css-validator/check/referer" target="_top">
        <img style="border:0;width:88px;height:31px" src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
    </div>

  </body>
</html>
